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In their article An Alternative Approach: Using Panels to Survey College Students, Sarraf, Hurtado, 
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confidence. 
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Abstract 

Eight short surveys based on select 
items from the National Survey of 
Student Engagement (NSSE) were 
administered to approximately five 
hundred students over a nine-week 
period at five diverse colleges and 
universities. The goal of the experiment 
was to investigate what impact a 
survey panel data collection approach 


would have on recruitment, survey data 
quality indicators, and scale properties. 
Results indicated higher response rates, 
shorter survey duration, and minimal 
impact on scale factor structures. 
However, both cost of incentives 

and panel member attrition make 

this alternative survey method less 
attractive than it would be otherwise. 


INTRODUCTION 


Survey researchers in higher education 
are engaged in an uphill battle with 
declining student response rates. 
Student cynicism, survey fatigue, and 
disinterest pose a substantial threat to 
optimal survey participation (Porter, 
2005). Despite significant challenges 
with the collection of student 
feedback, demands for evidence- 
based decisions in higher education 
continue to increase (Zhang, 2010). 

In light of these conflicting trends, 
identifying potential alternatives 

to standard survey practices may 
prove useful for a variety of higher 
education constituents, including 
institutional researchers and higher 
education scholars. Some scholars 
have already seriously considered this 
issue. For instance, Stern, Bilgen, and 
Dillman (2014) have proposed survey 
panels—a group of individuals that 
have agreed to respond to multiple 


future survey completion requests— 
as a potential solution to declining 
response rates. Despite today’s varied 
approach to implementing a survey 
panel, researchers have typically asked 
a group of individuals the same set of 
questions at different points in time. 
Though some academic disciplines, 
government agencies, and businesses 
have used survey panels for decades, 
higher education researchers and 
administrators have not often 
employed them to help with their data 
collection needs (Zhang, 2010). 


As a provider of assessment data to 
hundreds of colleges and universities 
across North America, and having 
witnessed its own response rates 
decline over the past decade, 

the National Survey of Student 
Engagement (NSSE) experimented 

with survey panels at five diverse 
institutions in spring 2014. The 

panel administration included eight 
surveys, with eight to ten items each, 
administered over nine weeks. Using 
standard NSSE administration results 

as a benchmark for each institution, 
this study documents the impact that 
using survey panels has on recruitment, 
survey data quality indicators, and scale 
reliability and factor structure. 
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SURVEY PANEL 
HISTORY 


For decades, survey panels have 

been used in market and medical 
research (Callegaro et al., 2014; 
Callegaro & Diogra, 2008). These survey 
administrations have historically 

been conducted in person, by phone, 
and by mail, and have recently 
transitioned to the Internet (Callegaro 
et al., 2014). Callegaro et al. estimate 
that the first online survey panel 

was administered in the mid-1980s 

in Europe. About a decade later, this 
method became popular in the United 
States. In particular, annual conferences 
dedicated to public opinion and 
survey research professionals, such 

as the one held for the American 
Association for Public Opinion Research 
(AAPOR), are rife with presentations 
from for-profit, not-for-profit, and 
government organizations (AAPOR, 
2015). According to Callegaro et al., 

the benefits of online survey panels 
are threefold: (1) quick data collection, 
(2) low administration cost, and 

(3) sampling efficiency. With a heavy 
reliance on online student surveys, 
these benefits may resonate with the 
needs of institutional researchers and 
higher education scholars. 


National education datasets developed 
by the federal government have used 
survey panels for better understanding 
the backgrounds and academic 
experiences of college students and 
their career trajectories. These include 
such studies as the Baccalaureate 

and Beyond and the Beginning 
Postsecondary Students Longitudinal 
Study (National Center for Education 
Statistics [NCES], n.d.). These survey 
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panels are distinct from many of 

the online survey panels previously 
discussed because they contact 
subjects only every few years, and not 
ona weekly or monthly basis. In terms 
of college student assessment, few 
researchers appear to be developing 
their own survey panels, although 
some have used them to answer 
methodological questions related to 
survey response behavior (Porter & 
Whitcomb, 2005; Sharkness & Miller, 
2014). 


Panel Definitions and Design 
Considerations 

Various researchers define survey 
panels differently, so it is important 

to clarify definitions. The traditional 
“panel” definition refers to a 
longitudinal survey panel that involves 
asking the same individuals the same 
questions across different points in 
time (Callegaro et al., 2014; Goritz, 
Reinhold, & Batinic, 2000; Hsiao, 2014), 
with each point in time referred to as 
a“wave.” This approach is inherently 
suited for studying particular changes 
among subjects. Many survey panels 
have evolved into access panels; an 
access panel is essentially a “database 
of potential respondents who declare 
that they will cooperate for future data 
collection if selected” (International 
Organization for Standardization [ISO], 
2012, p. 1). One form of panel survey 
utilizes a split questionnaire approach 
by dividing longer questionnaires into 
smaller ones (Raghunathan & Grizzle, 
1995). These various panel methods 
have been used in an effort to minimize 
missing data (Vriens, Wedel, & Sandor, 
2001). 


Types 

There are two primary survey panel 
types based on how members 

join: nonprobability panels and 
probability panels (Callegaro et al., 
2014). Nonprobability panels are 

open and allow anyone to volunteer 
to become a panel member. When 
members self-select into a panel, it is 
not possible to know the probability 
of selection, hence the nonprobability 
label. In contrast, individuals cannot 
join probability panels unless they 
have been invited to do so. Probability 
panels require that “all members of the 
population of interest have a known, 
non-zero probability of receiving an 
invitation to join” (Callegaro et al., 2014, 
p. 7). 


Sampling 

Segers and Franses (2014) found 

that it was difficult to survey every 
panel member at each point in time 
because of cost and the potential 

for nonresponse. To address these 
concerns, they utilized a rotating 
sampling method whereby panel 
members participated for a fixed 
amount of time with new panel 
members integrated at each wave. 
Other sampling methods include 
continuous, time, randomized, and 
matrix sampling. Continuous sampling 
means that the researcher surveys all 
individuals at each wave for the extent 
of the project. Time sampling refers 
to panel members being surveyed 

on a regular basis (e.g., biweekly) but 
not for every wave; they are rotated 
into waves, helping to ensure data are 
collected weekly, for instance, or on 
some other predetermined schedule. 
Randomized sampling refers to panel 
members being chosen at random for 


each particular wave. Finally, matrix 
sampling calls for a panel survey to 

be divided into groups of questions 
with panel members answering only 
one question group (Segers & Franses, 
2014). 


Incentives 

Incentives are frequently used in 
survey research in an effort to increase 
response rates, and panel studies are 
no exception. There are various types 
of incentives (e.g., guaranteed prepaid, 
guaranteed postpaid, and lottery). 

In relation to panel studies, there is 
conflicting evidence regarding the 
different incentive types and their 
effectiveness. Goritz (2006) found 

that cash lotteries (both one large 
prize and multiple smaller prizes) did 
not reliably increase panel response 
or retention rates. In a study on the 
impact of individual payment ona 
three-wave longitudinal experiment, 
Goritz, Wolff, and Goldstein (2008) 
found that guaranteed payments had 
a negative effect on the first wave and 
a positive effect on the second wave. 
Additionally, after a review of the 
literature, Callegaro et al. (2014) found 
that postpaid incentives are most 
impactful when the members have 
prior experience or knowledge of the 
organization administering the survey. 
None of these studies used a college 
student population so the current 
study should shed light on incentive 
effectiveness, and, more specifically, on 
guaranteed prepaid incentives. 


This study was based on a probability 
panel that used a split-questionnaire 
design with continuous sampling, and 
both guaranteed prepaid and lottery 
incentives. 


STUDY RATIONALE 
AND RESEARCH 
QUESTIONS 


Though aggregate response rates 

for NSSE have slowly declined for 
years (the 2014 administration being 

a recent exception), they are still 

high enough to reliably estimate 
institution-level engagement for the 
vast majority of participating schools. 
As a recent study using NSSE data has 
shown, low response rate estimates 

of student engagement are very 
similar to estimates based on high 
response rates (Fosnacht, Sarraf, Howe, 
& Peck, in press). Nevertheless, it is 
incumbent on NSSE and other higher 
education researchers to look closely 
at alternative data collection methods 
to see if there are more-effective 

ways to collect student feedback. 
Whether or not minimal response 
rates and respondent counts can 
estimate institution-level engagement 
sufficiently, low response rates matter 
to skeptical audiences who believe 
higher response rates are required 

for decision making. If response rates 
drop precipitously for a significant 
number of institutions in the future, 
those interested in student opinion 
may need to rely more heavily on other 
data collection methods, such as focus 
groups, nonprobability sample surveys, 
and/or survey panels to supplement 
their standard survey administrations. 


The following research questions guide 
this study and help us assess whether 
survey panels can serve NSSE and the 


wider institutional research community. 


Questions are grouped into three 
general areas for assessing the viability 


of panels: (1) recruitment, (2) survey 
data quality indicators, and (3) scale 
reliability and factor structure. 


Recruitment 

1. What percentage of students 
tried to register for the NSSE 
survey panel, and how do these 
rates compare to standard 
administration invitation response 
rates? 

2. What were claim rates for the 
panel’s guaranteed incentives? 

3. How do panel members compare 
demographically to non-panel 
members and to standard 
administration respondents? 


Survey Data Quality Indicators 

4. What are response rates to 
individual panel surveys, and how 
do they compare to standard 
administration rates? 

5. What are survey duration 
and completion results by 
administration type? 


Scale Reliability and Factor Structure 
6. Do panel and standard survey 
administration methods produce 
similar scale scores? 
7. Does the factor structure of scales 
vary by survey administration 
method? 


Study Background 

Data Source and Sample 

To address the research questions, 

we combined NSSE standard 
administration survey data with panel 
data for five colleges and universities. 
These institutions differed by size (total 
undergraduate enrollment), status 
(public or private), and designation 
(college or university).' To preserve the 
anonymity of the five study institutions, 
we have named them Small Private 
College, Small Private University, 
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Medium Public University, Medium 
Private University, and Large Public 
University. 


Each institution participated in the 
2013 standard administration with 
the exception of the Large Public 
University; that university’s data 
came from an experimental 2014 
standard administration that used 

a smartphone-optimized version of 
NSSE. All five institutions participated 
in a spring 2014 administration using 
survey panels. In fall 2013 NSSE 

staff contacted 12 diverse colleges 
and universities to offer them an 
experimental, no-cost survey panel 
administration during the following 
semester (spring 2014). NSSE staff 
selected five of the six institutions that 
expressed interest. 


Because of other survey commitments 
at the Large Public University, NSSE 
staff randomly selected 25% of all 
first-year and senior students to 
participate in either the panel or 
smartphone experiments, and then 
followed up by randomly assigning the 
selected students to either the panel 
or the standard administrations. The 
other four institutions provided all 
first-year student and senior records 
for sampling. Though not part of the 
original research design, the concurrent 
panel and standard administrations for 
the Large Public University allow for 
stronger claims about panels, whereas 
at the other institutions we compare 
results obtained at different times and 
from different student populations. 


We analyzed 3,331 respondents from 
both administration types: 2,832 
standard administration and 499 panel 
respondents. Of that total, 67% were 
female, 17% were underrepresented 
minority, and 6% were part-time 
students. With the exception of sex, 
these demographics reflect missing 
data for some institutions. We also 
used 12,272 first-year student and 
senior non-panel members from 

the 2014 panel administration to 
assess recruitment success and panel 
composition; this included 1,900 
students at the Large Public University 
assigned to the experiment, which 
represents a fraction of all its first-year 
students and seniors. 


Panel Administration Details 

Our goal was to recruit 50 first-year 
students and 50 seniors from each 
institution. We drew repeated random 
student samples by institution and 
class level in order to send panel 
registration invitation e-mails; we 

sent registration invitations to 6,595 
students, ranging from 650 to 1,950 per 
institution. Invitations emphasized that 
each of the eight surveys would take 
about one minute to complete. We sent 
students only one panel registration 
e-mail and concluded the registration 
process over about two days. Those 
that attempted to register after all one 
hundred openings had been filled were 
told that it would no longer be possible 
to join the project. As promised in the 
registration invitation e-mail, all panel 
members could immediately retrieve 

a $10 Amazon.com gift card using an 
online portal that was created for the 


study. Additionally, the registration 
e-mail informed students that the 
names of students who completed all 
eight panel surveys would be included 
in a drawing for one $250 Amazon.com 
gift card at each school. 


Upon registration, students received 
an e-mail requesting they complete 
the first survey, followed by a reminder 
e-mail two or three days later if they 
had not responded. Approximately 
every week we delivered another 
e-mail invitation and follow-up 
reminder for another panel survey. 
As the administration for each 
survey began, we also posted the 
survey link to the online portal. It 
took approximately nine weeks to 
administer all eight surveys. During 
the last week, a final reminder was 
delivered to students who had 

not completed all eight surveys, 
encouraging them to log in to the 
online portal to complete all surveys. 


Standard Administration Details 

As part of a standard NSSE 
administration, all first-year and senior 
students received survey recruitment 
messages sometime between 
February and May, and the survey 
officially closed on June 1. As with 

all institutions, NSSE sent students 

at the five study schools five e-mail 
recruitment messages. Institutions 
decided to use survey incentives 

to boost response rates and, when 
appropriate, they included related 
text in all recruitment messages. Each 
recruitment e-mail included a URL 
that linked to the online survey and to 


'The labels “small,” “medium,” and “large” signify that total undergraduate enrollment is either fewer than 5,000 (small), between 5,000 and 15,000 


(medium), or more than 15,000 (large). 
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Table 1. NSSE Panel Survey Content 


1 Collaborative Learning, Student—Faculty Interaction 8 
2 Reflective & Integrative Learning, Academic Major, Class Level 9 
3 Higher-Order Learning, Writing Practices, Age 8 
4 Quantitative Reasoning, Effective Teaching Practices, Parental Education 9 
5 Discussions with Diverse Others, Learning Strategies, Course Challenge, Living Location 9 
6 High-Impact Practices, Class Preparation, Reading 9 
7 Quality of Interactions, Number of Courses Taken (overall and online), Race 8 
8 Supportive Environment, Other Institution Experience 10 


the informed consent statement. The 
informed consent statement advised 
students that the survey would take 

a total of between 15 and 18 minutes 
to complete. Though we did not use 
available data for this study, institutions 
could append topical module item 
sets ranging in content from academic 
advising to civic engagement, and 
participate in a consortium that 
administered additional items. 


Survey Content 

The 2013 standard administration 
survey included approximately 104 
survey items.? The eight experimental 
panel surveys, however, had 70 items 
combined (see table 1). The panel 
administration incorporated arguably 
the most important topics from the 
standard instrument, including survey 
items from 10 primary scales used for 


official NSSE reporting and several 
other important student experiences, 
background, and demographic items, 
including academic major and parental 
education. Panel survey content 

order did not follow the standard core 
instrument's layout. For a complete 

list of panel survey items, readers may 
contact the authors at 
nsse@indiana.edu. 


STUDY FINDINGS 


Recruitment 

1. What percentage of students tried to 
register for the NSSE survey panel, and 
how do these rates compare to standard 
administration invitation response rates? 
After a single e-mail invitation, 
between 6% and 20% of invited 
students registered, or attempted to 
register, for the panel administration 


at the five participating institutions 
(see table 2). The proportion that 
responded varied significantly by 
institution, overall, and by class level 
(pp < .001). At all but one institution, a 
lower proportion of first-year students 
than seniors registered for the panel 
study. Differences in registration rates 
favored seniors by 2 to 8 percentage 
points. Compared to the standard 
administration response to the first 
recruitment message, interest in the 
panel was appreciably higher at all five 
institutions. The Small Private College 
showed the greatest difference with 

8 percentage points (12% standard 
administration rate versus 20% panel 
rate), while the Small Private University 
and the Medium Private University 
showed the least, with a 2 percentage 
point difference at each. 


?For a survey facsimile, see NSSE (n.d.b). 
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Table 2. Initial Response to Participation Request by Administration Type 


Standard Overall First-Year Seniors FY vs. Panel 
Administration? Students (SR) SR Gift Card 
(FY) (X?) Retrieval 
Rate 
Small Private College 12.0% 20.0% 17.4% 23.0% + 78.0% 
Small Private Univ. 10.0% 12.4% 9.4% 17.4% ae 75.0% 
Medium Public Univ. 2.0% 6.9% 6.0% 8.0% + 78.0% 
Medium Private Univ.° 4% 6.0% 6.9% 5.4% 79.8% 
Large Public Univ. 2.0% 9.3% 7.0% 11.4% oe 81.0% 
xX n/a KK KK KK 


Note: + p <.1;* p <.05; ** p < .01; *** p < .001 (X?-test) 


@ Standard administration results reflect response rate five days after initial invitation to complete survey. 


> Panel registration results reflect total registered panel members and unsuccessful registration attempts divided by the total number of registra- 
tion invitations delivered. Approximately 105 students across all five schools tried to register but were not included in the panel because of the 


100 student quota per institution. 


‘This institution offered an incentive for completing the standard administration survey. 


2. What were claim rates for the panel's 
guaranteed incentives? 

Overall, 78% of the 499 panelists 
claimed their $10 Amazon.com 

gift card. There was no statistically 
significant difference in claim rates 
among institutions (table 2). Claim 
rates were also unrelated to class 
level: 76.2% of first-years and 80.4% of 
seniors claimed their incentives. 


3. How do panel members compare 
demographically to non-panel members 
and standard survey administration 
respondents? 

Panel members appeared very 

similar to non-panel members 

at each institution for two of the 
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three demographic variables we 
analyzed: full-time enrollment and 
underrepresented minority status 
(see table 3). We found no statistically 
significant differences for these two 
demographic variables at the five 
institutions. However, for four of the 
five (the Large Public University being 
the exception), there was a 10 to 17 
percentage point greater proportion 
of females among panel members 
compared to non-panel members. 


In terms of these demographic 
variables, panel members were 
generally comparable to standard 
administration respondents at each 
institution, as well. We see a single 


statistically significant difference 

in the proportion of females by 
administration type across the five 
schools: 51% of the Large Public 
University’s panel members were 
female whereas 62% of its standard 
administration respondents were 
female. 


Survey Data Quality Indicators 
4, What are response rates to individual 
panel surveys, and how do they compare 
to standard administration rates? 

Panel survey response rates at the five 
institutions ranged from 95% for the 
Small Private College's first survey to 
71% for the Medium Private University’s 
seventh survey (table 4). All institutions 


Table 3. Panel Member Characteristics Compared to Non-Panel Members and Standard Administration Respondents by 


Institution 
Female Sig. Full-Time Sig. Minority Sig. 
Panel member 74% 99% 11% 
mall Priv ll 
ome ae Coles? | -iiadsanelmeniber 57% * 99% 10% 
78% 
Standard respondent ° 100% 9% 
Panel member 70% 97% n/a 
Small Private Univ. 
Non-panel member 57% fa 97% n/a 
Standard respondent 62% 97% 5% 
Panel member 63% 94% 18% 
Medium Public Univ. 
Non-panel member 53% i 90% 19% 
Standard respondent 65% 91% 18% 
Panel member 75% 91% 25% 
Medium Private Univ. 
Non-panel member 60% 2 89% 28% 
Standard respondent 71% 93% 24% 
Panel member 51% n/a n/a 
Large Public Univ. 
Non-panel member 49% n/a n/a 
Standard respondent 62% 7 n/a n/a 


Note: * p < .05 (X?-test) 


had response rates over 90% for 

the first survey; rates for the eighth 
survey ranged from 72% to 86%. With 
the exception of the second survey, 
statistical tests indicate that response 
rates to individual panel surveys do not 
vary across institutions. As is common 
with panel administrations, response 
rates declined across the eight panel 
surveys. Percentage drops in response 
rates from the first to the eighth survey 


ranged from 11% for the Small Private 
College to 21% for the Medium Private 
University. Standard administration 
response rates ranged from 34% for 
the Small Private College to 12% for 
the Large Public University. In all cases, 
response rates for panel surveys far 
exceeded final rates for corresponding 
standard administrations at each 
institution. 


5. What are survey duration and 
completion results by administration 
type? 

Analysis showed that all panel surveys 
averaged a total of 9.2 minutes to 
complete, compared to 11.7 minutes 
for standard survey core items. If a 
panel member completed one item 

in a survey, he or she almost always 
completed all the items. 
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Table 4. Response Rates by Administration Type 


Small Privat 
ata © 95% 93% 85% | 85% 84% | 84% 84% 86% 100 34% 333 
Small Private 

ae 92% 83% 80% 77% 78% 78% 78% 78% 100 28% 341 
Medium 

angen 92% 88% 84% 83% 81% 80% 79% 76% 100 15% 753 
re 

Leena 93% 78% 79% 74% 72% 73% 71% 72% 99 21% 718 
Large Public 

es 91% 83% 81% | 80% 77% 78% 77% | 75% 100 12%? 308 
x2 * n/a 


Note: * p < .05 (X*-test) 
@ Institution offered an incentive for responding to the standard administration survey invitation. 


© Institution offered incentive after initial invitation; a random sample of students was used for this school’s administration. 


Table 5. Average Number of Panel Surveys Completed and Survey Completion Rates by Administration Type 


Small Private College 7.1 81% 73% 

Small Private Univ. 6.8 73% 81% + 
Medium Public Univ. 6.7 76% 77% 

Medium Private Univ. 6.3 66% 80% es 
Large Public Univ. 5.6 72% n/a 

x + " o 


Note: + p <.1;** p <.01 (X*-test) 
@Completion rate defined as the percentage of respondents completing 90% of all survey items. 
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However, not all panel members 
completed every survey. Panel 
members completed an average of 

6.5 out of 8 surveys with notable 
variation across institutions (table 5). 
Students at the Large Public University 
completed 5.6 surveys on average, 
whereas those at the Small Private 
College completed 7.1. 


Completion rates, defined as the 
proportion of students completing 

at least 90% of standard NSSE items 
or all survey panel items combined, 
favored the standard administrations 
of the Small Private University and the 
Medium Private University by 8 and 
14 percentage points, respectively. In 
contrast, the Small Private College and 
the Medium Public University showed 
no statistically significant differences. 
We could not reliably calculate the 
Large Public University completion 
rates for the mobile-optimized 
experimental administration due to a 
programming error. 


Scale Reliability and Factor 
Structure 

Knowing that a panel administration 
approach would not substantially 
change psychometric scale properties 
is very important to assessing overall 
panel viability. In order to shed light on 
this issue, we answered two questions 
using NSSE scales, otherwise known 
as engagement indicators (Els). El 
scores measure the frequency with 
which students engage in various 
educationally enriching behaviors, 
students’ perceptions of campus 


support, and the quality of students’ 
interactions with different groups, such 
as faculty and other students. 


6. Do panel and standard survey 
administration methods produce similar 
scale scores? 

To assess the reliability of El scores, 

we compared these scores by 
administration type for both first-year 
students and seniors. Differences in 

El scores might be attributed to the 
fewer number of panel respondents 
relative to the standard administration, 
or to unknown changes that occurred 
at campuses in the one-year lapse 
between administrations, although 

we would not expect meaningful 
differences unless an institution 
undertook major programmatic 
changes. To determine whether El 
scores differed by administration 

type, we first calculated effect sizes by 
subtracting the standard administration 
El score from the panel El score and 
dividing the difference by the standard 
administration El standard deviation. 
We then reported and evaluated the 
absolute values of the effect sizes by 
using guidelines developed by NSSE 
staff suggesting that an effect size of 

.1 is considered small, .3 is considered 
medium, and .5 is considered large 
(Rocconi & Gonyea, 2015). We used 
t-tests to determine whether any 
differences were statistically significant 
at the .1 alpha level after applying a 
correction for false discovery rates, 
which is a concern when conducting 
multiple tests concurrently (Benjamini & 
Hochberg, 1995). 


Our study suggests that both survey 
administration approaches generally 
yield similar results (table 6). Out of the 
100 comparisons, only nine showed 
both meaningful and statistically 
significant differences, ranging from 
.36 to .81 effect size. Most notably, 

the Large Public University, which 

had the methodological advantage 

of concurrent random assignment 

into either the panel or standard 
administration, showed no differences. 
For the 20 comparisons for each 
institution, we see anywhere between 
zero and three notable differences. 

For senior populations at the Small 
Private University and the Medium 
Public University we found two notable 
differences, which represents the 
greatest class-specific discrepancy 

for any institution. In terms of certain 
scales that may present particular 
challenges, Quantitative Reasoning 
appears to have the greatest difference 
between the two survey administration 
approaches. 


7. Does the factor structure of scales vary 
by survey administration method? 

We used multi-group confirmatory 
factor analysis to test for measurement 
invariance (or consistent factor 
structure) by the two types of survey 
administration methods. Confirming 
measurement invariance ensures 

that scores relate “to the same set 

of observations in the same way in 
each group” (Borsboom, 2006), which 
allows the researcher to reliably 

draw conclusions from intergroup 
comparisons. Based on multiple 


3 The 10 Els analyzed are (1) Higher-Order Learning, (2) Reflective and Integrative Learning, (3) Learning Strategies, (4) Quantitative Reasoning, (5) Col- 
laborative Learning, (6) Discussions with Diverse Others, (7) Student-Faculty Interaction, (8) Effective Teaching Practices, (9) Quality of Interactions, and 
(10) Supportive Environment. For more information about Els, see NSSE (n.d.a). 
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Table 6. NSSE Engagement Indicator Effect Sizes Comparing Panel and Standard Administration Scores 


Class Small Small Medium Medium Large 
reesl Engagement Indicator Private ae ublic paluie Lele 
College Univ. Univ. Univ. Univ. 
First-Year Higher-Order Learning 0.05 0.55 * 0.01 0.28 0.01 
Reflective & Integrative Learning 0.15 0.28 0.18 0.16 0.08 
Learning Strategies 0.11 0.29 0.11 0.29 0.28 
Quantitative Reasoning 0.81 * 0.48 0.29 0.01 0.31 
Collaborative Learning 0.10 0.11 0.09 0.13 0.14 
Discussions with Diverse Others 0.24 0.07 0.13 0.23 0.22 
Student—Faculty Interaction 0.14 0.07 0.34 0.14 0.44 
Effective Teaching Practices 0.20 0.02 0.22 0.22 0.16 
Quality of Interactions 0.00 0.00 0.29 0.03 0.14 
Supportive Environment 0.04 0.01 0.45 * 0.14 0.05 
Senior Higher-Order Learning 0.01 0.03 0.28 0.24 0.01 
Reflective & Integrative Learning 0.08 0.02 0.19 0.06 0.19 
Learning Strategies 0.28 0.15 0.25 0.30 0.14 
Quantitative Reasoning 0.31 0.33 0.36 * 0.42 * 0.02 
Collaborative Learning 0.14 0.46 * 0.00 0.14 0.16 
Discussions with Diverse Others 0.22 0.03 0.08 0.21 0.01 
Student—Faculty Interaction 0.44 * 0.10 0.18 0.00 0.09 
Effective Teaching Practices 0.16 0.24 0.32 0.30 0.21 
Quality of Interactions 0.14 0.53 * 0.09 0.10 0.09 
Supportive Environment 0.05 0.09 0.52 * 0.22 0.13 


Note: * p < .1 (t-test) 
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confirmatory factor analysis model 
results, we grouped each of the 10 
scales for both class levels into one 

of five increasingly invariant (or 
consistent factor structure) categories: 
(1) variant, (2) configural invariance, 
(3) weak factorial invariance, (4) strong 
factorial invariance, and (5) strict 
factorial invariance.* In order for NSSE 
to use survey panels for measuring 
any specific El score in the future (and 
to compare these results to standard 
survey administration results), strong or 
strict factorial invariance for each scale 
by class level would be needed. ° 


Our results indicated that 8 out of 10 
first-year student Els met the criteria 
for strict factorial invariance. Higher- 
Order Learning and Supportive 
Environment met the lesser criteria 
for strong factorial invariance. Senior 
year results indicated that five Els 

had strict factorial invariance while 
three had strong factorial invariance. 
Higher-Order Learning and Supportive 
Environment results indicated variant 
factor structures between panel and 
standard administrations, which 
would make any inter-administration 
comparisons untenable. 


LIMITATIONS, 
DISCUSSION, AND 
CONCLUSIONS 


Limitations 

There are several limitations associated 
with this study’s design and analyses. 
First, results for four of the five study 
institutions came from comparing 
potentially different student populations 
during the spring 2013 and spring 2014 
semesters, which could have unknown 
effects on study outcomes. Second, 

we guaranteed that all panel members 
would receive a $10 Amazon.com gift 
card incentive as well as a chance to win 
a $250 Amazon.com gift card, whereas 
standard administration sample members 
at three institutions did not get any 
incentive and the other two conducted 
lotteries. Making any firm conclusions 
about the relative attractiveness of 
panels is therefore confounded; the 
incentives could have impacted data 
quality outcomes, especially because 
they relate to survey completion. 
Related to this point is whether any 
selection bias exists given that panel 
members chose to join the panel after 
just one registration e-mail. Although 
we reviewed several key demographics 
and determined that panel members 
were generally representative, they 
may still be different in unknown ways 
that influence engagement results. 
Finally, given the relatively small 
number of panel respondents at each 


institution, we could not analyze results 
by subgroups. This study’s results 

may not offer sufficient information 

for those who typically review survey 
results by academic major fields or 
other subpopulations. 


Discussion 

This study provides various insights 
about college student survey panels, 
one possible alternative to the standard 
survey administration approach used 
by NSSE, other national survey projects, 
and many institutional research offices. 
Foremost among our findings, panel- 
registered students from all five study 
institutions responded to panel surveys 
at very high rates (70% or more), far 
above response rates for corresponding 
standard administrations. These 
findings support the idea of Stern et al. 
(2014) suggesting survey panels may 
be one solution for declining response 
rates. 


As social exchange theory might 
suggest, these high response rates 
likely stem from the minimal cost of 
participation (cost for each survey was 
about a minute answering relatively 
innocuous questions) in relation to 
several perceived benefits, including 
the incentives offered and the 
opportunity for students to provide 
helpful feedback about their college 
experience. Another valid explanation 
is that students may feel a strong 
ethical obligation to participate 


* To accomplish this categorization, models were run separately for each administration type until the same model fit both groups well. If no model fit 
both groups, we rejected measurement invariance and pursued no additional testing. Assuming a model fit both groups well, we then ran tests for the 
four types of invariance sequentially. Once a lower level of invariance was tested and rejected, we did not proceed with running tests for higher levels of 
invariance. Criteria used for determining acceptable model fit were RMSEA < .06, Chi-square p-value > .05, and CFI/TLI > .90. Strict factorial invariance 
required chi-square difference test p-values greater than .05 and ACFI values of less than .01. 


5 Fora detailed review of multi-group confirmatory factor analysis methods, see Little and Slegers (2005). 
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after receiving their $10 gift card. A 
guaranteed incentive of this amount 
is generally unheard of among NSSE 
standard administrations and may not 
be a particularly realistic amount for 
some institutions to offer students, 
especially if they want more than 100 
panel members. For this reason, we 
encourage others to experiment with 
lowering the incentive's dollar value 
to minimize expenditures but without 
sacrificing too much on response rates. 
The survey methodology literature 

on college student panel incentives is 
thin, if not entirely missing, so further 
investigation is certainly warranted. 


With the exception of gender, we 

also found panel members to be 
similar to non-panel members and 
standard administration respondents 
using full-time enrollment and 
underrepresented minority status. 
Females traditionally respond to NSSE 
survey requests at higher rates, so 
their overrepresentation among panel 
members is not especially surprising. 
These results bode well for the use of 
panel studies since administrators may 
not have to worry excessively about 
self-selection bias. 


Consistent with similar studies 
completed in market and medical 
fields, results indicate, unsurprisingly, 
that panel studies do offer a rapid way 
to collect information from students. 
Panel members answered each of the 
eight- to ten-item surveys in about a 
minute with nearly zero item skipping 
or nonresponse. The short duration 
and practically nonexistent item 
nonresponse are other reasons that the 
panel survey approach may be a viable 
alternative to standard administration 
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practices. This study's conclusions differ 
from the conclusion of Apodaca, Lea, 
and Edwards (1998) that individuals are 
reluctant to participate in surveys with 
multiple components. 


In terms of NSSE scales, the vast 
majority of results indicated scores 
originating from panels are comparable 
to those from standard administrations, 
even with a limited number of 
respondents. With the exception of two 
Els for seniors, scale factor structures 
do not appear to be affected by the 
data collection method, something 
that would be very important for 
analyzing panel and standard 
administration results longitudinally. 
The reasons for Higher-Order Learning 
and Supportive Environment scales 
failing invariance testing are unclear at 
this time. At a minimum, this finding 
would complicate combining results 
from these two different survey 
administrations. 


In contrast to these favorable panel 
results, this study also highlights 
several issues of concern with 

using survey panels. Based on our 
recruitment findings, it does not 
appear that students perceive panel 
membership to be significantly 

more attractive than a standard 
administration approach, even with 
the $10 guaranteed Amazon.com gift 
card and promises of a minute-long 
survey each week. There are several 
possible explanations, such as students 
weighing the protracted involvement 
over nine weeks against the incentives 
provided and determining that the 
costs actually outweigh the benefits. 
Second, we saw that panel member 
attrition drove overall missing data 


levels for some institutions to a 

point that resembles or surpasses 
standard administration results. 
Though it is challenging to generalize 
based on our results, when we look 

at the percentage of students that 
complete 90% of all survey items by 
administration type we can reasonably 
conclude that completion rates are 
not always improved by dividing up a 
longer survey. Variations of this study's 
panel design, such as rotating panel 
members to reduce fatigue, may yield 
more-promising results in terms of 
missing data levels. 


Amidst a growing need for survey data 
to inform decisions in higher education, 
students are at increased risk for survey 
fatigue. Survey panels may offer some 
relief by limiting the need to administer 
surveys to entire campus populations. 
While 100 participants joined the panel 
and received eight survey invitations, 
the vast majority of students at each 
institution did not receive more than 

a single registration e-mail, thus 
reducing any potential frustration with 
multiple requests. High panel response 
rates may effectively limit the need 

to send several reminders to sample 
members. Additionally, we provided 
students the option to go back and 
complete previous waves of the 

survey using an online portal, thereby 
eliminating the need for several survey 
reminder messages that could increase 
perceptions of burden and actual 
survey fatigue. 


Conclusions 

College and university students often 
perceive requests to complete a 
relatively long 15- to 18-minute survey 
as burdensome, especially considering 


the many other survey requests they 
receive. This overload of surveys 
challenges survey administrators to 
think about more-effective ways to 
collect student feedback than the 
customary cross-sectional survey 
design that NSSE and other survey 
research projects currently employ. 
Survey panels represent one data 
collection alternative that is worthy 

of further investigation: our results 
point to high response rates, short 
completion times, and minimal impact 
on measurement scales. Panel member 
attrition and associated missing data 
levels, however, makes this option 
potentially problematic. The price 

for encouraging panel membership 

by using incentives may be cost 
prohibitive to some as well, especially if 
the researcher needs significantly more 
panel members for analyzing campus 
subpopulations. Our hope is that others 
will start experimenting with panels to 
see if there are ways to address these 
issues. Obviously, we do not know 
what the future holds for the standard 
survey administration approach, but 
investigating some viable alternatives 
may prove helpful in the future. 
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